Software Research's TestWorks (STW(tm)) suite of tools improves and controls software quality through automated testing.
TestWorks consists of three product lines that work independently or as an integrated tool suite to provide an efficient, automated testing environment.
STW/Regression(tm) is the most complete solution for automating the execution and verification of tests. It is used for initial testing and retesting to verify that modifications have not caused unintended impact.
The test suite manager, SMARTS(tm), allows tests to be executed according to an easy-to-use hierarchical structure and then conditionally evaluates based on the verification criteria selected for each test.
The capture/playback tool, CAPBAK(tm), records mouse movements, keyboard activities, and widget calls into a "C" language test script that can be played back later. It also has bundled capabilities for load generation for client-server applications.
The comparison tool, EXDIFF(tm), compares bitmap image or ASCII value files. Extraneous discrepancies can be discarded during the differencing process.
STW/Coverage(tm) uses state of the art compiler technology to measure how well test cases exercise programs at the unit, system and integration test levels. It provides logical branch (C1), function-call pair (S1) and path class (Ct).
The logical branch coverage analyzer, TCAT(tm), measures the number of times each branch has been exercised for both True and False conditions. The results are displayed in coverage reports and annotated directed graphs.
The function-call pair coverage analyzer, S-TCAT(tm), measures the number of times each function-call has been exercised, as errors in parameters are extremely common. The results are displayed in coverage reports and annotated call-trees.
The path coverage analyzer, TCAT-PATH(tm), measures the number of times each derived path class (sequence of branches) in a module was exercised. Results are displayed in coverage reports and directed graphs.
The test data observation tool, T-SCOPE(tm), dynamically displays logical branches and function-calls as they are being exercised. Results are shown in call-trees, directed graphs and slider bars.
STW/Coverage is available to support programs written in C, C++, Ada, and FORTRAN.
STW/Advisor helps the developer, tester and manager answer questions such as where additional testing resources should be applied, which parts of the code are not portable, and where engineering resources should be applied for rewriting or restructuring the source code.
The metric product, METRIC(tm), analyzes C, C++, Ada or FORTRAN source and calculates the Halstead Software Science Metrics to measure data complexity, the Cyclomatic Complexity Metrics to assess logic complexity, and basic size metrics.
The static analysis product, STATIC(tm), provides detailed syntax and semantic information for C programs. STATIC performs more detailed analysis than compilers, processing an entire program or individual files and generating a report that warns of any anomalies in the code that are likely to contain bugs, be non-portable, or extraneous. Analysis results are presented in an easy-to-read report.
The test data generator, TDGEN(tm), creates additional tests by substituting either random or sequential data values in an existing test script. TDGEN increases the size of a test suite with minimal work and limited resources, thereby more fully exercising the program-under-test.
************************
STW/Regression(tm) is the most complete test execution suite, designed to overcome the tedious and error-prone process of manual testing. Three components are included in STW/Regression: CAPBAK(tm) for automated capture and playback of user sessions, SMARTS(tm) for test organization and management, EXDIFF(tm) for test verification.
STW/Regression can be used for GUI Host Based Testing, Client/Server Testing, Terminal Emulation Testing, and Remote System Testing
CAPBAK records all user activities during the testing process including keystrokes, mouse movements, widget calls, and verification information. The captured images and characters provide baselines against which future test runs are compared.
CAPBAK has several modes for record/playback: True-Time automatically records keyboard and mouse activity into an "C" language test script and plays back the user input exactly as recorded. Character Recognition is available for adapting the test to minor application changes to the graphical user interface (GUI). Widget (object)-level testing is available for testing object calls independent of screen placement, color, or other minor variables.
CAPBAK/X can run multiple client-application tests on the same workstation for the purposes of load generation, performance assessment and multi-user test emulation.
SMARTS organizes and manages an extensive number of test scripts for the purpose of automating and allowing unattended testing. The ``test tree hierarchy'' emulates the modularity and functionality of the tested application. Test cases can be supplemented with activation commands, comparison arguments, system calls, evaluation methods, and control structures (if, else and while clauses). Other files can be incorporated, allowing for a modular organization of multiple test hierarchies.
EXDIFF verifies bitmaps captured during a recording session and automatically compares them with the actual images at playback. Character Verification is also supported: EXDIFF determines a successful test based on the actual values, regardless of differences in font and background color.
User-specified areas of an image or ASCII file can be ignored with EXDIFF's positive masking capability. Unrestricted mask creation allows unlimited, overlapping positive masks to be set for an individual image.
Reverse Masking enables inverse processing of an image file mask, where only the user-specified area is compared during the differencing process.
****************
CAPBAK/X(tm), one of the programs in STW/Regression (along with EXDIFF and SMARTS), is designed to overcome the tedious and error-prone process of manual testing by automating it. It captures into a "C" language test script all user activities during the testing process including keystrokes, mouse movements, widget calls and verification information. EXDIFF, one of CAPBAK/X's companion products, in turn supports Character Verification, determining a successful test regardless of differences in font or background color.
Future test runs then entail the playing back of these test sessions to determine if an application performs as expected. Each playback test session is executed as if the user is running the test. CAPBAK/X's automatic synchronization ensures a reliable playback, allowing tests to be run unsupervised as many times as the tester wants.
Response images and extracted characters corresponding to their baselines are automatically captured, with comparisons done on-the-fly in a batch process later.
Test sessions can also be played back on a "preview" display window. As a program traces through the recorded test script file, keyboard and mouse events are shown, allowing the user to relate an event with its corresponding recorded test script code. When an error is detected, the user can directly modify the recorded test script.
CAPBAK/X's editing utility inserts sleep commands or increases delays at button clicks and carriage returns, sets maximum and minimum delays on key clicks, pauses after certain defaults, strips extraneous mouse movements, and decomposes the recorded tests script into smaller sub-files.
With the help of its other STW/Regression(tm) companion tools, EXDIFF(tm) and SMARTS(tm), the testing process is completely automated. The SMARTS test management system organizes CAPBAK/X's test sessions into an hierarchical structure for execution individually or as a part of a test suite and then conditionally evaluates each test according to the verification method selected, whether True-Time, Character Recognition or widget mode. Discrepancies are reported by SMARTS for further analysis. Extraneous discrepancies, however, can be discarded in the comparison process via the EXDIFF product. SMARTS' test execution reports and statistics show the PASS/FAIL results for a test or a test suite.
A modified version of the X-server behaves just like a standard X-server, except it is simulated in memory and not tied to any physical screen. This allows playback activity to run in the background, freeing the screen to continue executing other processes or commands.
Virtual Playback plays back recorded keystrokes and mouse movements to the Virtual X-server. Screen image response files are saved to regular bitmap files that can be compared to the baseline images on a PASS/FAIL criteria.
CAPBAK/X can confirm tests on one machine that were initially created on a different machine with a different display size, allowing for portability.
CAPBAK/X works from a graphical user interface (GUI), and offers unique command line functionality. It also has the capability of issuing record/playback commands from configurable keyboard function keys.
*******************
STW/Regression/MSW(tm) is designed to overcome the tedious and error-prone process of manual testing by automating the execution, management and verification of a series of tests. Three components are included in STW/Regression: CAPBAK(tm) for automated capture and playback of user sessions, SMARTS(tm) for test organization and management, CBDIFF(tm), which is part of CAPBAK, for test verification.
CAPBAK records all user activities during the testing process including keystrokes, mouse movements, captured bitmap images, ASCII files, and widget (object) calls. The captured images and characters provide baselines against which future test runs are compared.
CAPBAK has several modes for record/playback: True-Time automatically records keyboard and mouse activity into an ASCII-format test script and plays back the user input exactly as recorded. Character Recognition is available for adapting the test to minor application changes to the graphical user interface (GUI). Widget (object)-level testing is available for testing object calls independent of screen placement, color, or other minor variables.
SMARTS organizes and manages an extensive number of test scripts for the purpose of automating the testing process. The Automated Test Script (ATS) ``test tree hierarchy'' emulates the modularity and functionality of the tested application. Test cases can be supplemented with activation commands, comparison arguments, system calls, evaluation methods, and control structures (if, else and while clauses). Other ATS files can be incorporated, allowing for a modular organization of multiple test scripts.
A detailed record of test outcomes and timing statistics is saved to a default log file which automatically generates a variety of reports. A status report lists the most recent test execution outcomes; a history report includes current and past test results for every test executed. A regression report lists only those tests whose outcomes have changed since previous test activation. A certification report summarizes the total number and percentage of tests that have passed and failed.
CBDIFF verifies bitmaps captured during a recording session and automatically compares them with the actual images at playback. With Character Recognition Verification, CBDIFF determines a successful test based on the actual values, extracting the values from bitmap images using Optical Character Recognition (OCR) and translating them into ASCII characters, regardless of differences in font and background color.
User-specified areas of an image file can be ignored with CBDIFF's positive masking capability. Unrestricted mask creation allows unlimited, overlapping positive masks to be set for an individual image.
Reverse Masking enables inverse processing of an image file mask, where only the user-specified area is compared during the differencing process.
*******************
CAPBAK/MSW(tm) helps you design and develop tests that automate the testing process. It captures all user activities during the testing process including keystrokes, mouse movements, widget calls and verification information into a "C" language script that is easily understood. The captured images provide baselines against which future reruns of the tests are compared.
Future tests then entail the playing back of these test sessions. CAPBAK/MSW's multiple synchronization modes ensure a reliable playback, so tests can be run unattended. Response images corresponding to baseline images are automatically captured. Comparison of the baseline and response images is done automatically and results are written to log file. The log file allows the tester quickly to identify where tests have failed.
The user has complete control over the tests. Both recording and playback sessions can be aborted, sped up, slowed down, or paused so the user can process other commands.
To verify tests have successfully played back, CAPBAK/MSW's CBVIEW util- ity displays test sessions' captured images. Tests are further verified with the CBDIFF utility which compares baseline and response file images for differences. CBDIFF's masking capability disregards those areas of images that are not necessary for comparison, such as time or date changes.
With the assistance of its other STW/Regression(tm) test management tool, SMARTS(tm), tests can be organized into an hierarchical structure for execution individually or as a part of a test suite and then condi- tionally evaluated according to the verification method selected.
CAPBAK/MSW has several capture/playback modes: True-Time, Character Recognition, and Widget level. With true time, the keyboard and mouse inputs are replayed exactly as recorded by the tester. Playback timing is duplicated from the server's own timing mechanism, allowing tests to be run as if executed by a real user. The results of the tests indicate any variances from the baseline cases, permitting the tester to determine the implication of those differences. Therefore, if a button moved to a different location in the window, it would be flagged as an error.
Character recognition allows the test to search for items that may have moved, or changed fonts since testing a previous version of the application. Character recognition helps extend the life of a test script by allowing it to adjust for minor changes in window layouts or fonts being used.
With widget playback, the X and Y coordinates on the screen are no longer significant as the application's widgets are activated directly. The same test script can run on multiple hardware and operating system platforms.
Each one of these modes is advantageous at different stages of testing during the software life cycle.
The verification method can include comparing saved images with the CBDIFF utility for small tests or comparing saved log file results of captured images for large tests.
Discrepancies are indicated for further analysis in SMARTS' PASS/FAIL test execution reports and statistics.
*****************
EXDIFF is a test evaluation facility that extends commonly available file comparison utilities. It works as a stand alone or within the STW/Regression(tm) tool set. File comparison includes ASCII and binary files, and bitmap image files saved either with SR's Capture/Playback tool, CAPBAK(tm) or the standard X Window Dump utility (xwd).
EXDIFF provides masking options which allow the user to specify areas within ASCII or image files to be ignored during the differencing process. While masking is concerned with positive masks (i.e. ignoring regions within specified boundaries), the EXDIFF inverse mask option enables negative masks to be differenced. In this mode, EXDIFF compares only the areas contained within the masked boundaries.
Xexdiff, the X Window system version of EXDIFF, performs a pixel-by-pixel comparison of the two saved image files and displays compared or differenced images to the screen. It indicates differences between image file colors; and it provides line and byte comparisons with the exdiff utility. Positive or negative mask coordinates are established in the image file using the Xmask utility.
The EXDIFF default mode exdiff differences two ASCII files as line-oriented or byte-oriented files. Results are reported in an easy-to-read format, which indicates the necessary standard line-editor commands to remove existing differences in the compared files. In this mode, EXDIFF is fully compatible with the standard UNIX diff utility. Using the EX utilities, similar ASCII files are reformatted for more reliable differencing with the exdiff utility.
With the help of its other STW/Regression companion tools CAPBAK and SMARTS(tm), the testing process is completely automated. The SMARTS test management system organizes test sessions recorded with CAPBAK into a hierarchical structure for execution individually or as part of a test suite according to the verification method selected. Typically, comparison image files represent baseline output versus newly generated outputs. Discrepancies are reported and stored by SMARTS for further analysis. Extraneous differences can be discarded in the comparison process via the EXDIFF tool. SMARTS automatically generates a variety of reports based on test results.
EXDIFF's masking capabilities are extensive: it can ignore user-specified areas of an image file during the differencing process with the Xexdiff utility; it allows unlimited, overlapping positive masks to be set for an individual image; it enables inverse processing of an image file mask, where only the user-specified area is compared during the differencing process; mask co-ordinates can be established either through the Graphical User Interface (GUI) or set in a configuration file to be read by EXDIFF at the start of execution.
EXDIFF allows trailing blanks (spaces and tabs) to be ignored and strings of blanks to be compared as equal; it reformats and/or divides ASCII files based on user-specified characters strings and text lines, preparing the files for more comprehensive differencing with the exdiff utility.
*******************
As a stand alone or within the STW/Regression(tm) tool set, SMARTS(tm) automates and simplifies the testing process by organizing the tests into a hierarchical tree, for the purpose of automating and allowing unattended execution of tests, and generating a variety of reports based on the test results.
SMARTS reads a user-designed "C" language test script. The hierarchically organized tests can be supplemented with test activation commands, comparison arguments and pass/fail evaluation methods. The test script can also be augmented with system calls to tools, such as the STW/Regression tool CAPBAK/X(tm), which replays captured user sessions. The test script automatically generates a directory type listing of tests, providing a means of interactively executing the test script or viewing results of previous runs.
All SMARTS commands are context sensitive. Test execution and reporting is based on the selection, either of an individual test or group of tests, from the displayed test script "test tree hierarchy". When executed, SMARTS performs the pre-stated actions, runs a difference check on the outputs against the baseline, and accumulates a detailed record of the test results. Using the STW/Regression comparison system EXDIFF(tm), differencing capabilities can be extended to ignore specified character strings and text line differences in ASCII files and masked areas in image files.
Four reports can be generated from test outcomes: the Status report lists the most recent test execution outcomes; the History report includes current and past test results for every test executed; the Regression report lists only those tests whose outcome has changed, identifying bugs which have been fixed or introduced since previous test activation; and the Certification report summarizes the total number and percentage of tests that have passed and failed, providing an overview of testing status.
SMARTS facilitates the testing process by providing an effective means of automatically determining discrepancies and monitoring the effects of regression testing.
SMARTS organizes and manages an extensive number of test scripts for the purpose of automating the testing process. The test script "test tree hierarchy" emulates the modularity and functionality of the tested application. It allows test cases to be supplemented with activation commands, comparison arguments, system calls, evaluation methods, and control structures (if, else and while clauses).
SMARTS will also incorporate other test script files, allowing for a modular organization of multiple test scripts.
Based on minimal test case information, a test generation utility expedites test script creation by generating a hierarchical SMARTS test structure consisting of an unlimited number of groups, sub-groups and test cases.
SMARTS allows unattended test processing by automatically executing all or a selected groups of tests. It runs tests according to criteria such as: executing tests a specified number of times, executing only new tests, running only tests which have failed during the previous execution, failing tests which exceed a pre-determined execution time and executing tests until a failed outcome.
*********************
STW/Coverage(tm) delivers the next generation of coverage analyzers. Using recursive descent compiler technology, STW/Coverage is able to efficiently handle a wide variety of language lialects. STW/Coverage give a numerical value to the completeness of a set of tests; it also show what parts of an application have been tested so that effort can be focused on creating test cases that will exercise the parts that were not previously tested.
STW/Coverage measures runtime coverage at the logical branch level for unit and system testing, measuring the number of times each segment off all branches has been exercised; call-pair coverage for integration and system tests measures the number of times each function-call has been exercised, as errors in parameters are extremely common; and path analysis for critical functions measures the number of times each path, which is a sequence of segments, was exercised.
STW/Coverage's analyzers gather usage statistics on programs, as they are being exercised, and create coverage reports. STW/Coverage generates graphs which reveal the control-flow structure of a module and call-trees which show the caller-callee relationship of a program. These displays are annotated by colors so the tester can very quickly determine what is and what is not being exercised in a set of tests.
Source viewing allows you to navigate from a graph to a function-call or logical branch in the source code.
STW/Coverage provides the only complete coverage solution available for test case validation, and provides coverage for mission-critical applications, such as an application needing FAA or FDA certification.
STW/Coverage allows for the instrumentation of isolated modules or exclusion of modules that have already reached a certain level of coverage; it also easily integrates instrumentation steps into your existing make file.
STW/Coverage provides support for host based as well as embedded and remote system testing. Software Research, Inc. customers are using our coverage tools on everything from host based DOS and UNIX platforms to remote testing on IBM AS/400s to embedded applications such as medical instruments and flight controls.
With T-SCOPE, you can monitor the executed program parts by generating logical branch and call-pair coverage data in real time. It shows coverage obtained for logical branches through directed graphs, call-pairs through call-tree displays, and an overall percentage through slider bars.
STW/Coverage works with multiple user interfaces: Graphical User Interface (GUI), Command Lines, and Menus. The tools in STW/Coverage are available to support programs written in C, C++, Ada and FORTRAN.
********************
STW/Coverage(tm)'s TCAT(tm) coverage analysis identifies which parts of a program's code has been tested at the individual function level.
TCAT measures the number of times logical branches have been exercised for both True and False conditions using its C1 metric, which is a more comprehensive yet efficient method than statement coverage. Statement coverage indicates which statements in a program have been exercised. This is a common coverage strategy, but does not guarantee that the program has been thoroughly exercised as there can be multiple actions in a single line, not all of which may be exercised even though the line is reported as tested.
TCAT instruments the program under test, placing markers at each logical branch. When test cases have been run against the instrumented code, the C1 metric collects data on the markers and stores the data in a trace file. TCAT then extracts the trace file information to create coverage reports, which indicate which code remains untested or frequently tested and which test cases duplicate coverage. In addition to coverage reports, TCAT also creates an archive file which stores all cumulative test information.
During the instrumentation process, directed graphs that show a module's control-flow structure are created. These graphical representations are one of TCAT's fundamental advantages: they are easy to interpret and give the user a quick overview of the results prior to looking at detailed reports.
The directed graphs allow the user to graphically view a logical branch's level of coverage with its unique color annotation feature and then displays and highlights the source code for that logical branch. The directed graphs can be used to identify unexercised code and to study the architecture of a module to aid in re-designing new test cases for unexercised branches.
TCAT provides support for host based as well as embedded and remote system testing. Software Research, Inc. customers are using our coverage tools on everything from host based DOS and UNIX platforms to remote testing on IBM AS/400s to embedded applications such as medical instruments and flight controls.
TCAT reports past, current and cumulative test results, providing hit, not hit, newly hit, newly missed, and linear and logarithmic histogram reports.
Using a number of graphs and reports, you can analyze code coverage with a variety of options: directed graphs use graphic and color overlays to diagram a module's logical branch relationship between decision points (nodes); different color overlays based on lower and upper threshold values indicate if a logical branch has been unexercised or has been heavily executed. In addition, you can easily navigate from a graph to a logical branch in your source code, making location of problem code much simpler.
TCAT works in multiple interfaces: graphical user interface (GUI), command lines, and menus. It is available in C, C++, Ada and FORTRAN. Using recursive descent compiler technology, TCAT and its companion products in the STW/Coverage bundle can efficiently handle a wide variety of language dialects.
**********************
STW/Coverage(tm)'s TCAT(tm) tool quantitatively measures logical branch and path completeness at the individual function level. Once individual modules and paths have been tested, system interface coverage testing should be performed to determine if all the interfaces have been exercised. TCAT's companion tool, S-TCAT, delivers this function.
The S1 measure expresses test effectiveness as the percentage of function-calls (this is every function, not just one per function) exercised in a program by a set of tests, relative to the number of such function-calls that exist in the system. S1 identifies the system interface errors -- which are almost always self-evident when a system's caller-callee relationships are actually exercised.
S1 coverage requires comprehensive tests that tend to remove a high percentage of latent software defects. On average S1 coverage adds 25% more defect detections to unit test level coverage.
S-TCAT(tm) instruments the system under test, placing markers at each function-call. When test cases have been run against the instrumented code, the S1 metric collects data on the markers and stores the data in a trace file. S-TCAT then extracts the trace file information to create coverage reports that show which calls remain untested, unused, or frequently tested and which test cases duplicate coverage. Along with coverage reports, S-TCAT also creates an archive file which stores all cumulative test information.
Instrumentation also generates call-trees that represent the caller-callee structure of a program. Call-trees aid the user in understanding code because they organize and identify a program's modules and connecting function-calls in a hierarchy.
The call-tree shows a function-call's level of coverage completeness with its annotation and color overlays. When a function-call obtains low coverage, the user can navigate to its source code. Along with coverage, call-trees display subtrees of caller-callee dependencies relative to a specific module and generate directed graphs for individual modules for an understanding of a module's control-flow structure. By using S-TCAT to monitor a system's level of testing, the user is assured that the proper modules and interactions are actually tested.
S-TCAT allows you to instrument isolated modules, or exclude modules that have already reached a certain level of coverage. It easily integrates instrumentation steps into your existing make file; S-TCAT also recognizes and processes all control structures.
S-TCAT works with Graphical User Interface (GUI), command lines, and menus. S-TCAT supports programs written in C, C++, Ada, and FORTRAN. Using recursive descent compiler technology, S-TCAT and its companion programs in the STW/Coverage product bundle can efficiently handle a wide variety of language dialects.
The STW/Coverage programs provide support for host based as well as embedded and remote system testing. Customers of Software Research, Inc. are using our coverage tools on everything from host based DOS and UNIX platforms to remote testing on IBM AS/400s to embedded applications such as those used in medical instruments and flight controls.
***********************
TCAT-PATH's path coverage analysis can be used to extend TCAT(tm)'s logical branch coverage at the unit test level. A path is a sequence of logical branches that can occur in a module. Most modules generally have a large number of paths. For this reason path coverage analysis is ordinarily limited to a few key modules.
To effectively deal with the large number of paths TCAT-PATH(tm) can group paths into equivalence classes, including multiple instances of iterations. By grouping paths into equivalence classes, TCAT-PATH's Ct metric then becomes more rigorous than TCAT's C1 metric.
While the C1 metric determines how many logical branches in module are exercised, the Ct metric determines how many of the possible path combinations in a module are exercised.
Obtaining reasonable coverage with a set of path tests can take 8 to 10 time as much work as it takes to obtain 100 percent branch coverage, but the result is much more effective. Defect detection efficiencies at 90 percent or better have been observed, even when path coverage is limited to 75%.
TCAT-PATH instruments the program under test, placing markers at each path. Paths for a selected modules are then generated, along with statistics about the paths. When test cases have been run against the instrumented code, the Ct metric collects data on the markers and stores the data in a trace file. TCAT-PATH then extracts the trace file information to a create hit/not hit coverage report for the selected module.
If the coverage report reveals paths that were not exercised, TCAT-PATH extracts and displays the logical conditions of the paths, providing the user with the information needed to design test cases that exercise the unexercised paths. The path conditions may reveal an invisible path, in which case the path can be deleted. When new test cases have been designed or invisible paths have been deleted, a new coverage report is generated.
Statistics are provided on the generated paths that includes information on the number of nodes, logical branches, the Cyclomatic Complexity number, number of paths, path length, and iteration groups.
Paths are represented graphically with directed graphs, which use graphic and color overlays to diagram a path. These directed graphs can be customized to make viewing easier and more relevant. Statistics are available which summarize information about the displayed module, including the number of nodes, logical branches, paths, and iteration groups and the cyclomatic complexity number.
TCAT-PATH supports path coverage analysis for C, C++, Ada, and FORTRAN; it will work for Graphical User Interface (GUI), command line, and menu-driven programs. Using recursive descent compiler technology, the products in the STW/Coverage bundle can efficiently handle a wide variety of language dialects. TCAT-PATH and its companion products provide support for host based as well as embedded and remote system testing. Customers of Software Research, Inc. are using our coverage tools on everything from host based DOS and UNIX platforms to remote testing on IBM AS/400s to embedded applications such as those used in medical instruments and flight controls.
***********************
T-SCOPE(tm) works directly with STW/Coverage(tm)'s TCAT(tm) and S- TCAT(tm) tools to dynamically show the user how thoroughly test cases actually exercise an instrumented program. TCAT (Test Coverage Analysis Tool) measures C1 (logical branch) coverage for detailed unit testing. S-TCAT (System Test Coverage Analysis Tool) measures S1 (call-pair) coverage for system integration testing.
TCAT and S-TCAT instrument source code to place markers at each logical branch and call-pair. A directed graph that shows a module's control-flow structure, a call-tree that depicts an application's caller-callee relationship, and an object file are created from this instrumentation process. When TCAT and S-TCAT are used during testing, they report coverage status after tests have been executed against the instrumented source code.
T-SCOPE, on the other hand, visually demonstrates logical branches and call-pairs as they are being exercised while tests are running.
T-SCOPE links the created instrumented object file to its supplied runtime module, creating an executable. Once an executable is created, the user can choose to select modules' directed graph displays and an application's call-tree graph. Slider bars can also be selected that will show the percentage of coverage achieved for individual modules or the entire program, as each test is run.
When the necessary displays have been selected, thresholds can be set to show during test execution if a logical branch or a call-pair has been executed above or below the set threshold values.
When an application is executed, T-SCOPE updates the directed-graphs and call-tree by highlighting the exercised logical branches and call-pairs. Color annotation shows which logical branches and call-pairs have been hit less than the minimum threshold value, exercised more than the maximum threshold value, or executed between the two thresholds. This annotation clearly identifies areas that have been heavily tested or untested. Slider bars are also updated to show the percentage of coverage achieved for a module or a program after each test case is run.
T-SCOPE is especially useful for finding out why certain program parts are not covered and why certain parts are heavily exercised by the established test cases. By running test cases while using T-SCOPE, the user is better able to detect a bug or to create a new test case to exercise the un-hit code.
***********************
STW/Advisor(tm) analyzes source code to provide measurements and reports that aid in the management of project development. Three components are included in STW/Advisor: METRIC(tm) for quantitative analysis, STATIC(tm) for semantic and syntax analysis, and TDGEN(tm) for test data/file generation.
METRIC analyzes C, C++, Ada or FORTRAN source and calculates the Halstead Software Science Metrics to measure data complexity, the Cyclomatic Complexity Metrics to assess logic complexity, and basic size metrics, such as number of lines, comments and executable statements.
METRIC'S output includes a full report, which provides a set of metrics for each of the modules in a given source file; a summary report; an exception report, which lists where the code exceeds user-defined metric thresholds; and Kiviat Charts.
Software Science metrics derive hybrid metrics that include program length, predicted length, purity ratio, volume, and effort.
Cyclomatic Complexity Metrics determine the control-flow complexity of a program, based on the number and arrangement of decision points within the code.
Extended Cyclomatic Complexity determine the control-flow complexity of a program, based on the decision-making predicates.
Size Metrics include number of lines of code, number of blank lines, number of comment lines and the number of executable statements.
C programs are often the source of obscure bugs; many compilers pass bugs as legal C statements. STATIC handles C's unique problems by providing detailed syntax and semantic error/inconsistency reports for C programs.
A message report displays the source-code line, the file name, the line number, and a brief message about the problem. This includes syntax and semantic messages, warning messages, information messages, and elective notes.
Reports can be configured to display only selected types of messages.
STATIC is more detailed than a compiler. It points out code that is legal C, but is probably not what was intended, such as assignment (=) versus test (==), use of bitwise operators (& or | versus && or !!), never executed code or an empty for loop.
STATIC also looks across all source files to see if variable references and function-call parameters match; it identifies non-portable constructs, identifying pointer/pointer mixing, different sizes for short, int and long, problems when char is used to hold a character, signed/unsigned quantities, character set differences, and identifier length.
In order to make up for limited resources and more fully test applications, TDGEN creates additional tests from existing tests.
A Template File describes how selected test data values are to be placed in a typical test file.
A Values File indicates the actual input test values, test value ranges or test value generation rules for data descriptors that appear in the template file.
TDGEN processes the values file by constructing a data table with field names as keys, then scans the template file for special syntax with identifier field names, and finally substitutes values from the data table associated with corresponding field names. TDGEN selects test values from the data table sequentially, randomly, specifically, or calculates the total number of combinations possible.
***********************
STW/Advisor(tm)'s METRIC(tm) tool makes the best of limited time by com- puting software metrics for source code to quantitatively determine the code's quality. This information identifies overly complex modules requiring extra testing or rewriting or determines the number of man- hours necessary to develop a related or similar software package.
The most common measure of software complexity has been the computing of the number of lines of code. Although this measure provides an adequate estimate of program size, it does not necessarily predict program com- plexity. True software complexity measurements assign quantitative values to programs based on the structure of the code. These measures characterize various aspects of the code, which may be difficult to comprehend, modify or test.
METRIC offers a variety of software measurements. After METRIC processes a source code file, it automatically computes several metrics for the program. These metrics include the Halstead Software Science metrics which measure data complexity in routines, the Cyclomatic Com- plexity metrics which measure logic complexity in routines, and size metrics, such as number of lines, comments and executable statements. Metrics are reported in both configurable, easy-to-read reports and charts: the Full report provides a set of metrics for each of the modules in a given source file. the Summary report provides metrics for the program as a whole. The Exception report lists where the code exceeds user-defined metric thresholds. It advises whether each procedure and function falls within acceptable levels for your organization. The Kiviat charts display metric results from the Summary report, each metric is represented by an axis, and results are plotted with reference to user-definable upper and lower bounds. Because Kiviat charts indicate whether source code falls above or below set metric values, they serve as an invaluable means of quickly viewing the metrics to focus on for a particular program.
These reports and charts do not require an in-depth knowledge of metrics and computation.
Because METRIC identifies the most complex program modules, testing resources can be concentrated on just those identified complex parts.
Halstead Software Science metrics measure data complexity in routines. n1 determines the number of unique operators in a program (e.g. keywords); n2 determines the number of unique operands in a program (e.g. data objects); N1 determines the total number of operators; N2 determines the total number of operands. The above metrics derive hybrid metrics that compute program length, predicted length, purity ratio, volume, and effort.
The Cyclomatic Complexity metric determines the control-flow complexity of a program, based on the decision-making predicates; Extended Cyclomatic Complexity determines the control-flow complexity of a program, based on the decision-making predicates.
Size metrics measure number of lines of code, the number of blank lines, the number of comment lines, and the number of executable statements.
METRIC supports programs written in C, C++, Ada and FORTRAN.
***********************
STATIC(tm) performs a more detailed error check than a compiler, including locating non-portable constructs. It also looks at the entire program to find inconsistencies across the modules that comprise an application. This is especially important in multi-programmer projects.
STATIC processes a source code file or multiple files and generates a report of more than 300 possible syntactical, warning and informational messages.
STATIC has been designed to cope with the potential avalanche of messages. The report can be easily customized to fit organization unique programming styles with the following types of options: control options enable and/or disable messages individually, or in groups, by symbol name, or by type difference; flag options handle data types and syntax errors; size options sets the sizes of various scalars (short, int) for a target machine; compiler options disregard nonstandard constructs.
STATIC allows programmers to decide on the degree of checking that is necessary and focus on a single type or error, if they wish.
Static detects an extensive list of bugs: inter-module type inconsistencies; function parameter mismatches in number or in type, in the same module or across multiple modules; value return inconsistencies for functions (information is gleaned from return statements, calls and, optionally, declarations); used but not initialized errors (for auto, static and global variables, arrays and structs); declared but not used; used but not defined; unreachable code; assigned but not accessed (for auto, static and global variables, arrays and structs); unusual uses of Boolean as in:
flags & 4 == 0
(the programmer thought that & had a higher precedence than ==; it does not);
constant Booleans as in:
if ( x = 0 ) ...
(this ex-Pascal programmer thought he was testing for x being 0);
indentation checking; pre-processing anomalies; suspicious uses of semi-colons as in:
if (a>b);
strict and loose enumeration checking; order of evaluation errors, as in:
a[i] = i++;
wide variety of loss of precision errors such as int to char featuring our exclusive precision checking; excessive shift values; loss of sign; suspicious casts; mixing signed and unsigned quantities; comment within comments; pointer irregularities such as an increase in pointer capability and returning pointers to auto; ANSI `Quiet Changes'; unusual compile-time objects, including macros, typedefs, declarations, structs, unions, enums; header files not used; externals that can be made static; declarations that can be off-loaded from headers; name clashes within first count characters; one declaration hiding another; strong type checking options support full or partial typedef-based type checking.
***********************
STW/Advisor(tm)'s test data generator, TDGEN(tm), provides the capabil- ity to take an existing test script and substitute new data values to create additional tests, increasing the size of a test suite with minimal work in order to more fully exercise the program-under-test.
TDGEN produces an input test data file based on the following user created files: o A template file describes how selected test data values are to be placed in a typical test file through data descriptors. This file provides an outline of the eventual output file. o A values file indicates the actual test values, test value ranges or test value generation rules for each data descriptor.
According to user specification, such as sequential or random genera- tion, TDGEN replaces the template file's data descriptors with the value file's actual values. The descriptors in the template file are denoted by a special syntax; all other text, except for comments, is reproduced verbatim.
TDGEN behaves as an intelligent macro processor. The user either creates new test data files or configures existing test scripts to sub- stitute different data items for selected fields. With TDGEN, tens or hundreds of additional tests can be created in a short amount of time.
A Template File describes how selected test data values are to be placed in a typical test file. A Values File indicates the actual input test values, test value ranges or test value generation rules for data descriptors that appear in the template file.
TDGEN processes the values file by constructing a data table with field names as keys, then scans the template file for special syntax with identifier field names, and finally substitutes values from the data table associated with corresponding field names.
The types of values available include: enumerated values with no size restrictions; expanded values that include the first and last number in a sequence and are automatically expanded; and built-in values that can be integers, ASCII and real (floating) numbers.
Values are generated in four ways: Summary Mode, which calculates the total number of combinations possible with all fields in a given specification; Random Mode, which selects field values at random; Sequential Mode, which selects field values sequentially; and Specific Mode, in which the user indicates a specific order of data to be generated with choices of location-specific data, uniformly distributed data, or value-factored data.
The test data file can be customized with variable width, fixed width, left-adjusted or right-adjusted.
TDGEN is language-independent, and can be used with Graphical User Interfaces (GUI), command line, and menu-driven programs. It is available on single-user, multi-user, and site license basis.